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1 BACKGROUND OF THE INVENTION 

2 The present invention relates to in-place differential compression, where a body of data T is 

3 compressed with respect to a body of data S by performing copies from S in such a way that no 

4 additional memory is used beyond what is needed to store the longer of S or T\ that is, when 

5 decoding, S is overwritten from left to right as T is constructed in its place. 

6 There have been many patents and technical papers that pertain to data compression. Many relate 

7 to techniques different than ones that employ string copying such as Huffman coding (e.g., U.S. 

8 patent 4,646,061) or arithmetic coding (e.g., U.S. patent 4,905,297). Many relate to techniques 

9 that employ string copies but in a traditional data compression model where a single body of data 

10 is compressed, not in-place differential compression of a first body of data with respect to a 

11 second body of data; for example, U. S. patents such as Holtz [4,366,551], Welch [4,558,302], 

12 Waterworth [4,701,745], MacCrisken [4,730,348], Miller and Wegman [4,814,746], Storer 

13 [4,876,541, 5,379,036], Fiala and Greene [4,906,991], George, Ivey, and Whiting [5,003,307, 

14 5,016,009, 5,126,739], Rubow and Wachel [5,023,610], Clark [5,153,591, 5,253,325], Lantz 

15 [5,175,543], Ranganathan and Henriques [5,179,378], Cheng, Craft, Garibay, and Karnin 

16 [5,608,396], and technical articles such as Lempel and Ziv [1977, 1979] and Storer [1978, 1988, 

17 1982, 2002]. 

18 There have also been a number of patents and technical papers relating to differential 

19 compression that do not perform decoding in-place; for example: Squibb [5,479,654, 5,745,906], 

20 Morris [5,813,017], Muralidhar and Chandan [6,233,589], Thompson, Peterson, and 

21 Mohammadioun [6,671,703], and technical articles such as Weiner [1973] (who developed a 

22 linear time and space greedy copy/insert algorithm using a suffix tree to search for matching 

23 substrings), Wagner and Fischer [1973] (who considered the string-to-string correction problem), 

24 Heckel [1978] (who presented a linear time algorithm for detecting block moves using longest 

25 common substring techniques), Tichy [1984] (who used edit-distance techniques for differencing 

26 and considered the string to string correction problem with block moves), Miller and Myers 

27 [1985] (who presented a comparison program for producing delta files), Fraser and Myers [1987] 

28 (who integrated version control into a line editor so that on every change a minimal delta is 

29 retained), Reichenberger [1991] (who presented a greedy algorithm for differencing), 

30 Apostolico, Browne, and Guerra [1992] and Rick [1995] (who considered methods for 
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1 computing longest common subsequences), Burns and Long [1997b] (use delta compression to 

2 modify ADSM, Adstar Distributed Storage Manager of IBM, to transmit compact encodings of 

3 versioned data, where the client maintains a store of reference files), Hunt, Tichy and Vo [1998] 

4 (who combine Lempel-Ziv type compression and differential compression to compute a delta file 

5 by using a reference file as part of the dictionary to compress a target file), Factor, Sheinwald 

6 and Yassour [2001] (who present a Lempel Ziv based compression with an extended dictionary 

7 with shared data), Shapira and Storer [2002] (who give theoretical evidence that determining the 

8 optimal set of move operations is not computationally tractable, and present an approximation 

9 algorithm for a block edit-distance problem), Agarwal, Amalapurapu, and Jain [2003] (who 

10 speed up differential compression with hashing techniques and additional data structures such as 

1 1 suffix arrays). 

12 There has also been commonly available software available for differencing that does not 

13 employ in-place decoding with string copying, such as the UNIX diff 9 xdelta and zdelta utilities. 

14 Burns and Long [1997], M. Ajtai, R. Burns, R. Fagin, and D. D. E. Long [2002], and the U.S. 

15 patent of Ajtai, Burns, Fagin, and Stockmeyer [6,374,250] use a hash table with Karp-Rabin 

16 footprints to perform differential compression of one file with respect to another, using constant 

17 space in addition to that used by both files, but do not provide for in-place decoding. 

18 Burns and Long [1998], Burns, Stockmeyer, and Long [2002], and the U.S. Patent of Burns and 

19 Long [6,018,747] present an in-place reconstruction of differential compressed data, but do not 

20 perform the reconstruction with copies that overwrite from left to right. They begin with a 

21 traditional delta file and work to detect and eliminate write-before-read conflicts (increasing the 

22 size of the delta coding). 

23 The invention disclosed here is in part motivated by the research presented in Shapira and Storer 

24 [2003]. 



-3 - 



1 References, U.S. Patent Documents: 

2 4,366,551, "Associative Memory Search System", INVENTOR(S): Klaus E. Holtz, FILED: 

3 10/22/1979, GRANTED 12/28/1982 (continuations from 6/16/1975). 

4 4,558,302, "High Speed Data Compression and Decompression Apparatus and Method", 

5 INVENTOR(S): T. A. Welch, FILED: 6/20/1983, GRANTED: 12/10/1985. 

6 4,646,061, "Data Communication with Modified Huffman Coding", INVENTOR(S): R. E. 

7 Bledsoe, FILED: 3/13/1985, GRANTED: 2/24/1987. 

8 4,701,745, "Data Compression System", INVENTOR(S): J. R. Waterworth, FILED: 3/3/1986, 

9 GRANTED: 10/20/1987. 

10 4,730,348, "Adaptive Data Compression System", INVENTOR(S): J. E. MacCrisken, FILED: 

1 1 9/19/1986, GRANTED: 3/8/1988. 

12 4,814,746, "Data Compression Method", INVENTOR(S): V. S. Miller, M. N. Wegman, FILED: 

13 8/11/1986, GRANTED: 3/21/1989. 

14 4,876,541 , "System for Dynamically Compressing and Decompressing Electronic Data", 

15 INVENTOR(S): J. A. Storer, FILED: 10/15/1987, GRANTED: 10/24/1989. 

16 4,905,297, "Arithmetic Coding Encoder and Decoder System", INVENTOR(S): G. G. Langdon 

17 Jr., J. L. Mitchell, W. Pennebaker., FILED: 1 1/18/1988, GRANTED: 2/27/1990. 

18 4,906,991, "Textual Data Substitution Data Compression with Finite Length Search Windows", 

19 INVENTOR(S): E. R. Fiala, D. H. Greene, FILED: 4/29/1988, GRANTED: 3/6/1990. 

20 5,003,307, "Data Compression Apparatus with Shift Register Search Means", INVENTOR(S): 

21 G. A. George, D. L. Whiting, FILED: 10/6/1989, GRANTED: 3/26/1991. 

22 5,016,009, "Data Compression Apparatus and Method", INVENTOR(S): G. A. George, G. E. 

23 Ivey, D. L. Whiting, FILED: 1/13/1989, GRANTED: 5/14/1991. 

24 5,023, 610, "Data Compression Method Using Textual Substitution", INVENTOR(S): A. K. 

25 Rubow, D. R. Wachel., FILED: 6/13/1990, GRANTED: 6/1 1/1991. 

26 5,126,739, "Data Compression Apparatus and Method", INVENTOR(S): D. L. Whiting, G. A. 

27 George, G. E. Ivey, FILED: 1 1/27/1990, GRANTED: 6/20/1992 (Division of Ser. No. 

28 297,152, 1/13/1989, U.S. Patent 5,016,009). 

29 5,153,591, "Method and Apparatus for Encoding, Decoding and Transmitting Data in 

30 Compressed Form", INVENTOR(S): A. D. Clark, FILED: 7/4/1989. 



-4- 



1 5,175,543, "Dictionary Reset Performance Enhancement for Data Compression Applications", 

2 INVENTOR(S): C. B. Lantz, FILED: 9/25/1991, GRANTED: 12/29/1992. 

3 5,179,378, "Method and Apparatus for the Compression and Decompression of Data Using 

4 Lempel-Ziv Based Techniques", INVENTOR(S): N. Ranganathan, S. Henriques, FILED: 

5 6/30/1991, GRANTED: 6/12/1993. 

6 5,253,325, "Data Compression with Dynamically Compiled Dictionary", INVENTOR(S): A. D. 

7 Clark, FILED: 12/8/1989, GRANTED: 10/12/1993. 

8 5,379,036, "Method and Apparatus for Data Compression", INVENTOR(S): J. A. Storer, 

9 FILED: 4/1/1992, GRANTED: 1/3/1995. 

10 5,479,654, "Apparatus and method for reconstructing a file from a difference signature and an 

1 1 original file", INVENTOR(S): M. Squibb, FILED: March 30, 1993, GRANTED: 

12 December 26, 1995. 

13 5,608,396, "Efficient Ziv-Lempel LZ1 Data Compression System Using Variable Code Fields", 

14 INVENTOR(S): Cheng, Craft, Garibay, Karnin, FILED: 2/28/1995, GRANTED: 

15 3/4/1997. 

16 5,745,906, "Method and apparatus for merging delta streams to reconstruct a computer file", 

17 INVENTOR(S): M. Squibb, FILED: November 14, 1995, GRANTED: April 28, 1998. 

18 5,813,017, "System and method for reducing storage requirement in backup subsystems utilizing 

19 segmented compression and differencing", INVENTOR(S): Morris Robert J. T. , 

20 FILED: September 3, 1996, GRANTED: September 22, 1998. 

21 6,018,747, "Method for generating and reconstructing in-place delta files", INVENTOR(S): R. 

22 C Burns, D. D. E. Long , FILED: 1 1/26/1997 , GRANTED: January 25, 2000. 

23 6,233,589, "Method and system for reflecting differences between two files", INVENTOR(S): B. 

24 Muralidhar, K. Chandan , FILED: July 31, 1998 , GRANTED: May 15, 2001. 

25 6,374,250, "System and method for differential compression of data from a plurality of binary 

26 sources", INVENTOR(S): M. Ajtai, R. Burns, R. Fagin, L. J. Stockmeyer, FILED: 

27 February 3, 1997, GRANTED: April 16, 2002. 

28 6,671,703, "System and method for file transmission using file differentiation", INVENTOR(S): 

29 P. F. Thompson, K. M. Peterson, S. Mohammadioun , FILED: June 22, 2001, 

30 GRANTED: December 30, 2003. 



-5- 



1 References, Other Publications: 

2 M. Ajtai, R. Burns, R. Fagin, and D. D. E. Long [2002]. "Compactly Encoding Unstructured 

3 Inputs with Differential Compression", Journal of the ACM 49:3, 318-367. 

4 A. Apostolico, S. Browne, and C. Guerra [1992]. "Fast Linear-Space Computations of Longest 

5 Common Subsequences", Theoretical Computer Science, 92: 1, 3-17. 

6 R. C. Burns and D. D. E. Long [1997]. "A Linear Time, Constant Space Differencing 

7 Algorithm", Proceedings IEEE International Conference on Performance, Computing, 

8 and Communications. 

9 R. C. Burns and D. D. E. Long [1997b]. "Efficient Distributed Backup and Restore with Delta 

10 Compression", Workshop on I/O in Parallel and Distributed Systems (IOPADS). 

11 R. C. Burns and D. D. E. Long [1998]. "In-place Reconstruction of Delta Compressed Files", 

12 Proceedings ACM Conference on the Principles of Distributed Computing (PODC). 

13 R. Burns, L. Stockmeyer, and D. D. E. Long [2002]. "Experimentally Evaluating In-Place Delta 

14 Reconstruction", NASA Goddard Conference on Mass Storage Systems and Technologies 

15 (MSST), in cooperation with the Nineteenth IEEE Symposium on Mass Storage Systems, 

16 http://romulus.gsfc.nasa.gov/msst/. 

17 G. Cormode, M. Paterson, S. C. Sahinalp and U. Vishkin [2000]. "Communication complexity 

18 of Document Exchange", Proceedings of the 11 th Symposium on Discrete Algorithms, 

19 197-206 

20 M. Factor, D. Sheinwald, and B. Yassour [2001]. "Software Compression in the Client/Server 

21 Environment", Proceedings Data Compression Conference, IEEE Computer Society 

22 Press, 233-242. 

23 C. W. Fraser and E. W. Myers [1987]. "An Editor for Revision Control", ACM Transactions on 

24 Programming Languages and Systems 9:2, 277-295. 

25 J. J. Hunt, K. P. Vo, and W. Tichy [1998]. "Delta algorithms: An Empirical Analysis", ACM 

26 Trans, on Software Engineering and Methodology 7, 192-214. 

27 D. Huffman [1952]. "A Method for the Construction of Minimum-Redundancy Codes", 

28 Proceedings of the IRE AO, 1098-1101. 

29 A. Lempel and J. Ziv [1976]. "On the Complexity of Finite Sequences", IEEE Transactions on 

30 Information Theory 22: 1 , 75-8 1 . 



-6- 



1 W. Miller and E. W. Myers [1985]. "A File Comparison Program", Software- Practice and 

2 Experience 15:11, 1025-1040. 

3 C. Rick [1995]. "A New Flexible Algorithm for the Longest Common Subsequence Problem", 

4 Proc. 6th Annual Symposium on Combinatorial Pattern Matching, Espoo, Finland, 5-7. 

5 D. Shapira, and J. A. Storer [2002]. "Edit Distance with Move Operations", Proceedings 

6 Conference of Combinatorial Pattern Matching (CPM), Springer, 85-98. 

7 C. Reichenberger [1991]. "Delta Storage for Arbitrary Non-Text Files", Proc. 3rd International 

8 Workshop on Software Configuration Management, Trondheim, Norway, 12-14. 

9 D. Shapira, and J. A. Storer [2003]. "In-Place Differential File Compression", Proceedings 2003 

10 Data Compression Conference (published March 25, 2003), IEEE Computer Society 

1 1 Press, pages 263-272. 

12 J. A. Storer and T. G. Szymanski [1978]. "The Macro Model for Data Compression", 

13 Proceedings of the 10th Annual ACM Symposium of the Theory of Computing, San 

14 Diego, CA, 30-39. 

15 J. A. Storer and T. G. Szymanski [1982]. "Data Compression Via Textual Substitution", Journal 

16 of the ACM 29:4, 928-951. 

17 J. Storer [1988]. Data Compression: Methods and Theory, Computer Science Press (a subsidiary 

18 of W. H. Freeman Press). 

19 J. A. Storer [2002]. "An Introduction to Data Structures and Algorithms", Birkhauser - Springer. 

20 W. F. Tichy [1984]. "The String to String Correction Problem with Block Moves", ACM 

21 Transactions on Computer Systems 2:4, 309-321. 

22 R. A. Wagner and M. J. Fischer [1973]. "The String-to-String Correction Problem", Journal of 

23 the ACM 21:1, 168-173. 

24 P. Weiner [1973]. "Linear Pattern Matching Algorithms", Proceedings of the 14 th Annual IEEE 

25 Symposium on Switching and Automata Theory, 1-11. 

26 J. Ziv and A. Lempel [1977]. "A Universal Algorithm for Sequential Data Compression", IEEE 

27 Transactions on Information Theory 23:3, 337-343. 

28 J. Ziv and A. Lempel [1978]. "Compression of Individual Sequences Via Variable-Rate Coding", 

29 IEEE Transactions on Information Theory 24:5, 530-536. 



-7 - 



1 BRIEF SUMMARY OF THE INVENTION 

2 Therefore, it is an object of this invention to perform in-place differential compression. With 

3 differential compression, a body of data T is compressed with respect to a body of data S. That is, 

4 an encoder and decoder have available identical copies of 5, and the encoder may copy 

5 substrings from S to form T, A pointer that copies a string w from S achieves data compression 

6 when the bits representing that pointer are fewer than the bits representing w. We use the 

7 variable n to denote the size of T and m to denote the size of S, Differential compression is in- 

8 place if the memory containing S is overwritten when decoding T so that at no time is more than 

9 a total of MAX{m,n) memory used, in addition to the constant amount of memory to store the 

10 program itself along with its local variables. 

11 It is another object of this invention to rearrange the substrings of S to better align S and Tto 

12 enhance the effectiveness of copying substrings from S to T in-place. 

13 An in-place differential compression method according to the present invention includes the 

14 steps of rearranging substrings of S to improve the alignment of substrings of T with substrings 

15 of S and decoding in-place by copying substrings of S to form portions of T in a way that 

16 overwrites S. 
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1 BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWING 

2 FIG. 1 shows encoding on the top and on the bottom the way in which in-place decoding 

3 overwrites memory. 

4 FIG. 2 shows the well known method of sliding window compression (prior art). 

5 FIG. 3 shows the of linking gaps when off-the-shelf compression follows aligned moves. 
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1 DETAILED DESCRIPTION OF THE INVENTION 

2 With differential compression, a body of data S of size m is compressed with respect to a body 

3 of data T of size n. That is, both the encoder and the decoder have a copy of S and then a new 

4 string T may be encoded and subsequently decoded by making use of S (e.g. copying substrings 

5 of S). 

6 There are many practical applications where new information is received or generated that is 

7 highly similar to information already present. When a software revision is released to licensed 

8 users, the distributor can require that a user must perform the upgrade on the licensed copy of the 

9 existing version. When incremental backups are performed for a computing system, differential 

10 compression can be used to compress a file with respect to its version in a previous backup, with 

11 respect to a similar file in a previous backup, or with respect to a file already processed in the 

12 current backup. Differential file compression can also be a powerful string similarity test in 

13 browsing and filtering applications. 

14 One of ordinary skill in the art can understand that there are many ways that compressed data 

15 may be transmitted from an encoder to a decoder, including on a communications link, over a 

16 wireless connection, through a buffer, by transfer to and from a storage device, etc., and that the 

17 form of compressed data transmission from the encoder to decoder does not limit the invention 

18 herein disclosed. 

19 Differential compression is in-place if the memory containing S is overwritten when decoding T 

20 so that at no time is more than a total of MAX{m,n) memory used. Of course, the decoder has 

21 stored somewhere the executable code for the decoding program (possibly in read-only memory 

22 or hardware), which not is part of memory used when we refer to the computation being in- 

23 place; that is, in-place refers to the amount of memory used above and beyond the program code. 

24 The program code may also make use of some fixed number of local program variables (indices 

25 of for loops, etc.) which are also not part of the memory used when referring to the computation 

26 as being in-place. We allow the encoder to use more memory if needed. The restriction that the 

27 decoder must operate in-place is desirable because it reflects practical applications where the 

28 decoder may have unknown or limited resources. 
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1 It is an object of this invention to perform in-place differential compression with methods that 

2 are both powerful (i.e., typically achieve high compression) and provide for fast and space 

3 efficient decoding. 

4 MAX and MIN Notation 

5 We use the notation MIN{x,y} to denote the smaller of x and y and MAX{x,y} to denote the 

6 larger of x and y. 

7 Big O Notation 

8 It is standard in the computer science literature to use Big O notation to specify how the amount 

9 of time or memory used by a method increases as a function of the input size. For two functions/ 

10 and g, both of which map non-negative integers to non-negative integers, f(x) is said to be 

1 1 0(g(x)) if there exist two constants a and b such that for all integers x>a, j{x) < b * g(x). For 

12 example, if a parameter K is chosen in such a way that K is 0(log 2 (MIN{m,n})), then it must be 

13 true that there exists two constants a and b which are independent of m and n {a and b remain the 

14 same no matter what the values of m and n are) such that for all values of m and n for which 

15 MIN{m,rc} > a, K<b * log 2 (MIN{m,n}). 

16 The notation 0(1) denotes a fixed constant. That is, fix) is 0(1) if there exists two constants a 

17 and b such that for all integers x > a, fix) < b\ if we let c be the constant equal to the maximum 

18 value of fix) in the range 0 < x < b, then^) < c for all integers x > 0. A big O constraint can be 

19 combined with other constraints. For example, for a parameter K, saying that "K < MIN{m,n} 

20 and K is 0(VMIN{m,n}) ,f means that although K may be chosen to be a function of m and n (i.e., 

21 K is larger when MIN{m,n} is larger), for all m and n K < MIN{m,n}, and also, there exists two 

22 constants a and b such that K<b* VMIN{m,n}, for all m and n for which MIN{/n,n} > a. 
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1 Memory 

2 We use the term memory to refer to any type of computer storage, such as the computer's internal 

3 RAM memory, a hard drive, a read-writable disc, a communications buffer, etc. We use the term 

4 character to refer to the basic unit of data to which compression is being applied to a body of 

5 data in some computer memory. A common example of a character is a byte (8 bits) or a set of 

6 items that are stored one per byte (e.g. 7-bit ASCII codes). However, all of what we describe 

7 applies as well to other types of characters such as audio, image, and video data (e.g., for audio 

8 data that is stored using 12 bits per sample, the set of possible characters is the set of 4,096 

9 distinct 12-bit values). 

10 Although it is common for a body of data to be stored in sequential locations of memory, it is not 

11 required for this invention, it could be that different portions of the data are stored in different 

12 portions of memory. We assume that there is a linear ordering to a body of data (a first character, 

13 a second character, a third character, etc.). If a character x occurs earlier than a character y in the 

14 sequence of characters that comprise a body of data, then we say x is to the left of y, and y is to 

15 the right of x. 

16 The size of a body of data is the number of characters that comprise it, and corresponds to the 

17 size of the memory used to store it (if the body of data resides in two or more disconnected 

18 portions of memory, the size of the body of data is the sum of the size of each of the portions of 

19 memory it occupies). 

20 When we refer to the amount of memory used when decoding for in-place differential 

21 compression of a body of data T with respect to a body of data 5, it is always understood that we 

22 are referring to the memory used to store portions of S and T, and that there may be an additional 

23 fixed amount of memory used to contain decoding instructions and local variables used for 

24 decoding (indices for loops, etc.). 
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1 Sliding Window Compression 

2 Referring to FIG. 2, to help describe our invention, we first review the well known method in the 

3 prior art of sliding window compression. The standard UNIX gzip utility is an example of a 

4 sliding window compressor / decompressor. Given a string, sliding window compression 

5 represents it with a smaller string by replacing substrings by a pointer consisting of a pair of 

6 integers (d,l) where d is a displacement back in a window of the last N characters and / is the 

7 length of an identical substring. The reduction in size achieved on a string depends on how often 

8 substrings are repeated in the text, how the pairs (d,l) are coded, etc. Typically "greedy" 

9 matching is used (always take the longest match possible). 

10 There are many ways that have been proposed to encode pointers. A simple method based on 



1 1 fixed length codes is: 

12 • Displacements are represented with flog 2 (A0l bits. 

13 • Lengths range from 3 to some upper limit MaxLength, which can be represented by the 

14 MaxLength-2 integers 0 through MaxLength-3, and use \\og 2 {MaxLength-2)\ bits. 

15 • An initial flag bit distinguish a pointer (to a substring of 3 or more characters) from a 

16 code for a single character, where the leading bit is a 0 if the next [~log 2 (A)l bits to follow 

17 are a character and a 1 if the next Tlog 2 (A0l + \\og 2 {MaxLength-2) \ bits to follow are a 

18 displacement and length, where A denotes the size of the input alphabet. 

19 For example, if #=4,095, MaxLength=lO y and we use the 128 character ASCII alphabet 

20 (A=128), a single character is represented with one byte (a flag bit and 7 ASCII bits) and a 

21 pointer uses two bytes (a flag bit, 12 displacement bits, and 3 length bits). 

22 Other methods may use variable length codes (some of which may be able to represent a string 

23 of two bytes with less than 16 bits). 
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1 It is also possible to employ an off-the-shelf variable length coder to encode fields of a pointer. 

2 For example, to compress data where very large matches may be present, one could use the 

3 following coding method for pointers that employs an arithmetic coder: 

4 • The pointer format begins with a control integer between 0 and 4 that indicates one of the 

5 following cases: 

6 Case 0: No displacement or length; the bits to follow indicate a single raw character. 

7 Case 1: Displacements <4,096 that point to a match of length < 256. 

8 Case 2: Displacements between 4KB and 36KB that point to a match of length < 256. 

9 Case 3: Displacements larger than 36KB that point to a match of length < 256. 

10 Case 4: Displacements larger than 36KB that point to a match of length > 256. 

11 • Separate off-the-shelf arithmetic encoders are used to encode control integers, the raw 

12 character for Case 0, and the length fields for Cases 1 through 3. 

13 • A fixed code of either 12 or 15 bits is used for the displacements for Cases 1 and 2. A 

14 fixed length l~log 2 (A0l code is used for the displacement fields of Cases 3 and 4 and for 

15 the and length field of Case 4. 
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1 The Present Invention 

2 A problem with previous methods for differential compression is their use of more 

3 computational resources (e.g., time and memory) than may be necessary and / or the need to 

4 sacrifice achievable compression in order to achieve acceptable resources. Here we propose a 

5 high performance fast method for performing differential compression in place. 

6 The solution proposed here uses the ln-Place Sliding Window (IPSW) method to compress a 

7 body of data Fof size n with respect to a body of data S of size m: 

8 IPS W Encoding Algorithm : 

9 Step 1 : Append T to the end of S. 

10 Step 2: Compress T with a sliding window of size MAX{m,n}. 

1 1 When the encoder slides a window of size MAX{m,n}, if m is equal to or larger than n, then only 

12 at the start can pointers reach all the way back to the start of S, and as we move to the right, more 

13 of S becomes inaccessible. If m is less than n, then after producing the first n-m characters of T 9 

14 it again becomes the case that pointers cannot reach all the way back to the start of S. In either 

15 case, by overwriting the no longer accessible portion of S from left to right, decoding can be 

16 done in-place, using MAX{m,n} memory (encoding may use ra+n memory). That is, each time 

17 an additional pointer is received by the decoder, it is decoded and the window slides right to add 

18 characters to the right of T and remove them from the left of S. If m < n, then decoding increases 

19 the size m of the memory to store S to a size n of memory to store T. If m > n, then after decoding 

20 is complete, T resides in what used to be the rightmost n characters of 5, and the memory for 

21 what used to be the first m-n characters of S can be reclaimed. 

22 Of course, the decoding may use a fixed amount of memory for the decoding instructions (which 

23 could be read-only memory or hardware) and a fixed amount of memory to contain a fixed 

24 number of local variables used for decoding (indices of loops, etc.). This fixed amount of 

25 memory is independent of m and n. Using big O notation, the memory used for decoding 

26 instructions is 0(1) and the number of local variables is 0(1). When we say that decoding can be 

27 done in-place using MAX{m,n} memory, it is always understood that the memory used for 

28 decoding instructions and local variables is in addition to this MAX{m,n} memory. 
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1 Referring to FIG. 1, depicted is encoding (top) and decoding (bottom) for when the size of T is 

2 50 percent larger than the size of S, and we are at the point when rc-m+jc characters of T have 

3 been encoded. The hatched region is all of S on the top and the remaining portion of S on the 

4 bottom. The lightly shaded region is the n-m characters of T that have already been encoded and 

5 decoded by the decoder without having to overwrite S. The dark shaded region is the portion of T 

6 that has already been encoded by overwriting the first x characters of 5. The remaining m-x 

7 characters of T have yet to be encoded, and hence the rightmost m-x characters of S are still 

8 available to the decoder. The decoder's window can be viewed as two pieces: the already 

9 decompressed portion of T (the lightly shaded and darkly shaded regions) that is to the left of the 

10 pointer and the remaining portion of S that is to the right of the pointer (but was to the left of the 

11 lightly shaded and darkly shaded regions when encoding). So for each pointer decoded, at most 

12 two string copies must be performed: a single copy when the match is contained entirely in S 

13 (hatched region) or entirely in T (lightly shaded and darkly shaded regions) or two copies when 

14 the match starts in S and ends in T (the encoder encodes a match that crosses the boundary 

15 between the hatched region and the lightly shaded region). Since in many practical applications 

16 matches that cross this boundary are really just "lucky" (i.e., we may be primarily looking for 

17 large matches to S and if they are not found then a match in Tthat is a shorter distance away), an 

18 alternate embodiment of this invention is to forbid copies that cross this boundary, in order to 

19 further simplify decoding. 

20 Any reasonable implementation of sliding window encoding can be used that allows for large 

21 matches. For example, the UNIX gzip utility that uses a window of size 32K and maximum 

22 match length 256; it is easy to modify the UNIX gzip utility to use a simple escape code for long 

23 pointers (e.g. reserve the longest possible length or displacement for the escape code) so that the 

24 same algorithm can be used for normal matches and the increased pointer length is only 

25 associated with large matches. 

26 Another object of this invention is, when in some applications it may be that there is some 

27 additional amount of memory K available, to take advantage of this additional memory to 

28 improve the amount of compression achieved. It could be that K is a relatively small amount of 

29 memory that grows larger as m and n grow larger, such as K being 0(VMIN{m,rc}) or even K 

30 being 0(log 2 (MIN{m,n})), or it could be that K is relatively large, but still not large enough to 
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1 retain a separate copy of S while decoding (that is, K < MIN{m,n}), such as 50 percent of the 

2 additional memory needed (that is, K = MIN{m,n}/2) or even 90% of the additional memory 

3 needed (that is, K = (9/10)MIN{m,n}). An additional amount of memory K can be utilized to 

4 potentially improve compression by lengthening the sliding window to size MAX{m,rc}+AT, thus 

5 delaying by an additional K characters the point at which S begins to be overwritten when 

6 decoding. That is, encoding begins with S as the rightmost characters of a window of size 

7 MAX(m,rc)+tf. 

8 In many practical applications, such as distribution of successive versions of a software 

9 application, S and T are highly similar and reasonably well aligned; that is, large matches 

10 between S and T occur in approximately the same relative order. In this case, IPSW can be a fast 

1 1 method that performs in-place as well as methods that are not in-place. 

12 Another object of this invention is to achieve good performance even when S and T are not well 

13 aligned, by preceding IPSW with an additional step to improve alignment. Poor alignment could 

14 happen, for example, with a software update where for some reason the compiler moved large 

15 blocks of code around. An extreme case is when the first and second halves of S have been 

16 exchanged to form T(S = uv and T = vu); to decompress T the IPSW algorithm overwrites u as it 

17 copies v, and then the ability to represent v with a single copy is lost. Rather than modify IPSW 

18 (which is already a very fast and practical method that suffices for many and perhaps most 

19 practical inputs), we propose a preprocessing stage for IPSW that moves substrings within S to 

20 better align S with T. The compressed format can incorporate a way for the decoder to determine 

21 whether preprocessing was performed (e.g., compressed data can be begin with an initial bit 

22 indicating whether preprocessing has occurred). The encoder can compress T with IPSW and 

23 compare that to compressing T with respect to S not in place (so at all times all substrings of S 

24 are available to be copied). If the difference is significant, this initial bit can be set, alignment 

25 preprocessing performed, and a list of moves prepended the normal IPSW encoding. The 

26 decoder can perform the moves and then proceed in-place with normal IPSW decoding. 

27 If the encoder determines that S and Tare not well aligned, then the goal of preprocessing for the 

28 IPSW algorithm is to find a minimal set of substring moves to convert S to a new string 5' that is 

29 well aligned with T. We limit our attention to moves that are non-overlapping, where the moves 



- 17- 



1 define a parsing of S and T into matched blocks, {b} i=] r , and junk blocks, {x i9 yi} M y 9 that is, 

2 S=x 0 'b^ 1 yx ] 'b a(2) 'X 2 -x r ^b a(r) 'X r and T=y 0 -b r y r b 2 y 2 -y r _ r b r y r When using the sliding window 

3 method, we would like to copy substrings of S only from the part that was not yet overwritten by 

4 the reconstruction of T. That is, we would like to perform only left copies, i.e., a copy (s„ d if /,) 

5 that copies a substring with / f characters from position s, to position d x that satisfies s, > d v 

6 We can improve upon the idea of greedy block-move edit distance computation proposed in 

7 Shapira and Storer [2002] by using two different kinds of rearrangements of the blocks. Moves 

8 rearrange the blocks {b& isL , so that they occur in 5 and T at the same order. Jigglings move the 

9 junk blocks of S backwards, so that, as a side affect, the matched blocks are moved forwards. 



10 Move Preprocessing Algorithm: 

11 Step 1: Find Non Overlapping Common Substrings (NOCS) of S and T from longest to 

12 shortest down to a minimum length stoplength. These are the {b;}, =/ r blocks. 

13 Step 2: Compute the minimum number of rearrangements (moves and jigglings) in S so 

14 that the blocks {b} i=] r are left copies within S and T. 

15 Step 3: Generate S' according to step 2. 

16 Step 4: Compute TPSW(S',T). 



17 Step 1 can be performed by repeatedly applying a longest common substring computation (see, 

18 for example, the text book An Introduction to Data Structures and Algorithms, by J. A. Storer, 

19 Birkhauser / Springer, 2002), or by using the linear time method of M. Meyerovich, D. Shapira, 

20 and J. A. Storer ("Finding Non-Overlapping Common Substrings in Linear Time", Technical 

21 Report CS-03-236, Comp. Science Dept., Brandeis University); minlength can be tuned for 

22 different applications, where a value of 256 has been found to be a good general practical choice. 

23 Since the encoder is not restricted to work in-place, generating S' m Step 3 can be done in linear 

24 time by copying the strings to a different location. Step 4 uses the fast linear time IPSW 

25 algorithm that has already been described as one of the objects of this invention. 

26 We now describe how Step 2 can be done in quadratic time of the number of blocks, which is in 

27 the worst case 0((n/ 'stop length) 2 ); in practice, the number of blocks is much smaller than Vn, and 

28 Step 2 works in linear time. The NOCS found in Step 1 are renamed using different characters, 
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1 and then the method of Shapira and Sorer [2002] can be performed to compute minimum edit 

2 distance with moves to attain the moved characters for the minimum cost which correspond to 

3 moving the NOCS. Our next goal is to produce a (source, destination) format for the NOCS 

4 moves to be sent to the decoder based on the character moves. For example, when dealing with 5 

5 NOCS renamed by {7,2,5,4,5} one possibility to obtain 12345 from 15432 by character moves is 

6 by moving 2, 3, and 4, where 4 is moved backwards. Another option for transforming 15432 to 

7 12345 is by moving 3, 4 and 5, where 4 is moved forwards. Each one of these solutions can be 

8 obtained from the dynamic-programming table from different optimal paths going from cell [r,r] 

9 (for integers 1 through r) back to cell [0,0], from which their alignment can be extracted. The 

10 source and destination positions depend on the order the moves are performed. Therefore, the 

1 1 encoder performs and reports one character move at a time, updating the source and destination 

12 locations, before proceeding with the next move. We define each item's destination to be the 

13 corresponding positions just after the last aligned item to its left. If there is no such item, the item 

14 is moved to the beginning of the array. The move causes a shift of all items to its right, and the 

15 item is marked as an aligned item after the move is performed. Let {6,} /= , r be a set of r matched 

16 blocks to be moved from source positions s s to destination position d h and let {x} i=] r be the 

17 'junk' blocks we wish to jiggle to the beginning of 5, so that all blocks {b} i=I „ r perform only left 

18 copies, i.e., > d L . After performing the block moves in S to obtain S\ we have S r - 

19 Xobj-Xj^bfXfX^b/x,, and T = yobryj'b 2 'y 2 -y r .fb r 'y r To see that it is always possible to perform 

20 jigglings so that the matched blocks become left copies consider the worst situation, where all 

21 blocks {b}^ r are shifted all the way to the right, without ruining their order obtained from the 

22 edit-distance algorithm. Since the space available is at least w, we are left with only left copies. 

23 Thus, in the worst situation we perform r-1 moves and r jigglings. We are interested in 

24 minimizing this number. Each block that was moved already in the edit-distance with move 

25 algorithm, is farther moved to the right, so that it is now adjacent to its right non-junk neighbor. 

26 These moves are done for free, since they can be moved to the final point directly. At each stage 

27 of the jiggling algorithm, when we reach an illegal copy (i.e., a right copy), we choose the 

28 longest junk block to its right, and move it to the left of the leftmost illegal copy, which results in 

29 shifting all other blocks to the right. 
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1 A further object of this invention is for the decoder to perform the move pre-processing in-place. 

2 Let S be a string of length ra, and x, y denote the source and destination locations of a substring s 

3 of S of length / to be moved. Since the problem is symmetric for left versus right moves, and 

4 since we can easily incorporate an offset, assume jc=0, y>0, and y specifies the character that 

5 follows s after the move (e.g., if S = abcdefgxyz, 1=3, and y=7, then abc is moved so that S = 

6 defgabcxyz). A naive algorithm, that uses 0(1) additional memory and 0(y 2 ) time, moves the 

7 characters of s individually, shifting the entire string between the source and destination 

8 locations at each step. 

9 For a more efficient algorithm, observe that each character of s goes a distance d = y-/, and the 

10 move is tantamount to rearranging positions 0 through y-1 by the permutation i (i+d) MOD y. 

11 A standard in-place permutation algorithm (e.g., see the book of Storer [2002]) starts with 

12 positions 0 through y-1 "unmarked" and then for each position, if it is unmarked, follows its 

13 cycle in the permutation and marks each position visited, for a total of 0(y) time and y additional 

14 bits of memory for the mark bits. Here, mark bits for only MIN{/,J} positions are needed, since 

15 every cycle passes through at least one of the positions 0 through M and at least one of the 

16 positions / through y-1. If / or d is 0(log(m)), then we use 0(1) memory under our model, since 

17 we are assuming at least enough memory to store 0(1) local variables, which are each capable of 

18 holding 0(log(m)) bits (e.g., for m < 4 billion, 32 bits suffice to store an index variable in a loop 

19 that counts from 1 to m). If an additional amount of memory K is available that suffices to store 

20 these bits, then we can employ this additional memory for this purpose. Otherwise, in 

21 0(y 1/2 log(y)) time we can test y, y-1, y-2, ... until we find the largest prime p < y\ that is, d m 

22 operations suffice to test, and approximately ln(y) positions are tested, where In denotes the 

23 natural logarithm (this follows from the classic the "Prime Number Theorem" which appears in 

24 many basic texts on number theory). Then, let d' = /?-/, and using no mark bits, move s most of 

25 the way by the permutation i {i+d') MOD p (since p prime implies that the permutation is a 

26 single cycle that we can traverse until we return to the start). Finally, after adjusting the offset so 

27 that s starts at position 0, move s to position y 1 = /+(y-p) with the permutation i (i+(y-p)) 

28 MOD y'; since (y-p) « ln(y) = 0(log(m)), again, this can be can be done with 0(1) additional 

29 memory in our model (i.e., we are assuming at least enough memory to store 0(1) local 

30 variables, which are each capable of holding 0(log(m)) bits). 
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1 Another object of our invention is to provide the possibility of using an existing "off-the-shelf" 

2 compression method after a set of strongly aligned moves has been performed on 5. We say that 

3 a set of substring moves performed on S to create a string S2 is strongly aligned if when we write 

4 S2 below S and draw a straight line from each character in S that is moved to where it is in S2, 

5 then no lines cross. 

6 We can perform all strongly aligned moves for a string of length m in a total of 0(m) time and 

7 0(1) additional memory, using only simple string copies (that may overwrite characters that are 

8 no longer needed). We scan twice, first from left to right to copy blocks that go to the left and 

9 then from right to left to copy blocks that go to the right. Let x h y t and l h l<i<n, denote the source 

10 and destination locations and the lengths of k aligned move operations for a string stored in the 

1 1 array A: 

12 for i:=l to n do if i is a source position such that x{>y i then 

13 for y:=l to l x do A\y x + j] := A[x { + j] 

14 for i:=n downto 1 do if i is a source position such that jc ( <y f then 

15 for y:=/j downto 1 do A[y { + j] := A[x { + j] 

16 Any reasonable encoder (which need not work in-place) to construct a sequence of aligned 

17 moves may suffice in practice for applications where large aligned moves exist. For example, a 

18 greedy approach can parse the text twice, using two thresholds: C, and C 2 , where C, is much 

19 larger than C 2 . First extract matches longer than C 7 , and then search between two subsequent 

20 aligned matches for matches longer than C 2 . 

21 Assuming that a move operation is represented by three components (source position, destination 

22 position, and length), after the preprocessing step is complete, and the decoder has performed all 

23 of the move operations, the gaps can now be filled in by running a standard decompressor on the 

24 remaining bits so long as we are careful during preprocessing to remember these positions. This 

25 can be done by linking them together as depicted in FIG. 3; the encoder can be implemented to 

26 insure that each gap is large enough to hold a pointer (e.g., 4 bytes to handle moves of 4 billion 

27 characters). From the point of view of the off-the-shelf decoder, a contiguous string is produced, 

28 which we just happen to partition to fill the gaps. 
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1 Another way to use the off-the-shelf compressor is to modify it slightly so that gaps are 

2 compressed in -their local context. For example, with a sliding window method, the window 

3 could be defined to go contiguously back to the left end of the current gap and then continue on 

4 back into the previous block. 

5 A third way to use the off-the-shelf compressor is to use a single pointer value as an escape 

6 (followed by a displacement and length) to a match into the decoded string rather than into some 

7 additional memory used by the off-the-shelf encoder / decoder that is in addition to the 

8 MAX{m,n} memory normally allocated for in-place decoding; for example, this would be 

9 another way to make use of an additional amount K of memory. 

10 Although this invention has been primarily described as a method, a person of ordinary skill in 

1 1 the art can understand that an apparatus, such as a computer, could be designed or programmed 

12 to implement the method of this invention. Similarly, a person of ordinary skill in the art can 

13 understand that there are many types of memory that can be used to implement memory for the 

14 purpose of this invention (RAM memory, hard drive, read-write disk, data buffers, hardware 

15 gates, etc.). 

16 Since certain changes may be made in the above methods and apparatus without departing from 

17 the scope of the invention herein involved, it is intended that all matter contained in the above 

18 description or shown in the accompanying drawings shall be interpreted in an illustrative and not 

19 in a limiting sense. 
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